There is the 'less-than-visible' web and then there is also the completely 'dark web'.

Site operators can attempt to make their sites 'invisible'. This means that the spiders which compile the links for the search engines will ignore some sites.

1. Site owners can tag their site as a temporary site, used for testing and 'hot backup', which will dissuade search engines from listing it.
2. Site owners can also stick some instructions in the robots.txt file which sits in most web roots to keep the search engines away.
3. At a page level the line:

   <meta name="robots" content="noindex"/> 

   invites robots (aka bots or spiders) to ignore the page.

All of these will decrease the visibility of a site or some pages. But they are still part of WWW. Therefore they are there to be discovered and can be read of a person types in the url.

However, there are a number of reasons why people might want to go further and make it effectively impossible to locate their site or to see any content. They achieve this by leaving this web and making a web of their own - a dark web.

Why cloak your site to make it invisible?

Most people are struggling to optimise their site and make them popular and get a high rating from the search engines. But if you belong to a group that is not actively recruiting, then you would rather that people didn't stumble across you as they browsed. Celebrities, and others in the public eye, might prefer to share their thoughts and plans with a select few. So they remain invisible to those who are not invited to join.

You cannot do this on the web - You can encrypt the content but that would just provide a challenge for some. The only option is to keep off the web or to move to a 'parallel universe'. The dark web makes this possible.

One criticism is that the hidden sites provide cover for perverts and terrorists. But they also allow those living in repressive regimes as well as legitimate agencies and organisations to communicate privately. However, there is a consensus that these invisible sites should be tolerated and it is believed that governments are one of the major users. But another reason for tolerating them because there is very little that can be done about them apart from closing down the Internet on which all webs communicate.

How does it work?

These 'dark' systems are actually returning to an early model of the web when users would contribute space on their server. This was the 'magic' of the early Internet as each computer acted as a switch to find a route and then forward messages.

In those early days, most of these computers were in academic and government organisations. They provided a system that was very robust and able to survive nuclear war; This is why most of the software to run the Internet was developed, and donated to the world, by the US Department of Defence. Nothing sinister about that as it was just a way to provide a robust and survivable communications network. I do not recall anybody predicting that the World Wide Web would be built on top of the communications system we know as the Internet. But I digress.

It is just some special software that enables any shared communication system to work. But the complex routing and repackaging of messages already makes it virtually impossible to trace users as messages bounce through any number of relays computers on its way to the recipient. The data on the dark web is also heavily encrypted and the users who provide their computer as a 'switch' (like a telephone exchange) have no direct control over the data sitting on their machine before it is passed on.

These sites are wholly legitimate and safe to visit, even if just to do some research for your next thriller. The challenge is to find them! Here are some starting points.

The systems that people use to create a peer-to-peer web.

TOR Project (The Orion Router)

"Tor is free software and an open network that helps you defend against a form of network surveillance that threatens personal freedom and privacy, confidential business activities and relationships, and state security known as traffic analysis."

"Tor protects you by bouncing your communications around a distributed network of relays run by volunteers all around the world: it prevents somebody watching your Internet connection from learning what sites you visit, and it prevents the sites you visit from learning your physical location. Tor works with many of your existing applications, including web browsers, instant messaging clients, remote login, and other applications based on the TCP protocol."

I2P has this as its mission statement:

"I2P is an anonymizing network, offering a simple layer that identity-sensitive applications can use to securely communicate. All data is wrapped with several layers of encryption, and the network is both distributed and dynamic, with no trusted parties."

Most people see the internet as a place to tell the world about what they are doing - The search engines have made the billions of pages retrievable. That is what most people want.

However, there are situations where you might want to operate privately. If that applies to you then it is something that you can legitimately achieve.

Quality v Quantity   The invisible web    Research  Advance search 

 ©WritersServices.com 2000-2011