Scan for working web addresses

[Bmore]

I have a partial web address (eg www.bigcompany.com/users/) I know that there are hundreds of pages starting with the above example with alphanumeric combinations following it. There are millions of possibilities but only about a hundred or so pages.
-Is there a program that will scan the beginning of the address to search for info on all possibilites?
-Is there something in my address bar I can type in that would indicate a wildcard and it would search for all sites?
-Is there a search engine that will search for all sites beginning with my example...
ANY HELP WOULD BE GREAT

[Ezekiel]

I have a partial web address (eg www.bigcompany.com/users/) I know that there are hundreds of pages starting with the above example with alphanumeric combinations following it. There are millions of possibilities but only about a hundred or so pages.
-Is there a program that will scan the beginning of the address to search for info on all possibilites?
-Is there something in my address bar I can type in that would indicate a wildcard and it would search for all sites?
-Is there a search engine that will search for all sites beginning with my example...
ANY HELP WOULD BE GREAT

Google allows the use of wildcard (http://www.google.com/support/bin/answer.py?answer=**78) characters (*), so if the sites are indexed by google, you can use a search query like this:

"inurl:www.website.com/ *" site:www.website.com/ *

Example:

"inurl:www.freewebs.com/ *" site:www.freewebs.com/ *

I think that will work, but I didn't try it. All you have to do is replace 'website.com' with the address of the hosting service, then it will find all the websites hosted by them.

[SyntaXmasteR]

First download wget

Then run this batch script. It will tell you which sites are active.



echo off
cls

set site=0
set limit=20
set counter=0

:CHECK

if %site% EQU %limit% GOTO :EXIT
wget -q www.bigcompany.com/users/%site%
if %errorlevel% EQU 0 echo %site% >> %userprofile%\desktop\SitesFound.txt&& set /a counter=%counter%+*
set /a site=%site%+*
goto :CHECK


:EXIT
cls
echo Number of sites found was %counter%
echo FINISHED
echo --------
pause


Set the Site to your starting point
Set the limit to your ending point

[Bmore]

Thanks!! I will try that.

[Bmore]

I downloaded wget.
I changed the website, www.bigcompany.com to the web address in which I was interested in the batch script. Then I copied and pasted into the wget command window. Nothing happened.
I must've missed something.

[SyntaXmasteR]

Yes, place wget in your system*2 folder. Now you can run the batch script.

...just realised what you did - you DO NOT copy and paste the code into a command window. You save the code in notepad as a .BAT file after you have place WGET in your system *2 folder. Double click the file to run

[Bmore]

Thanks again for the update..I will try it

[Bmore]

Thanks, I did get it to start searching pages starting with the beginning of a website address. However it started at /*, /2, /* etc..increasing * every time...However it does not seem to be able to search apla and numeric...EG
www.company.com/user/ar5t or /ij*r or /4rt5
Thanks though...I will have to take a different route