Bye Bye bing

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 12982
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Bye Bye bing

Postby AUbicycles » Fri Apr 20, 2018 6:22 am

Guess what - I think I finally managed to get bing to stop overloading this site.

They have, what could be described as, an unethical bot. This crawls the site and saves it for their search engine. Yes, bing is a search engine from microsoft. The problem was, it ignore the search engine instructions and furthermore would frequently overload the site. When it visited, the site slows down for everyone else as bing was going overboard.

Though I finally got a contact partner and although they wanted me to make the site vulnerable and share user data (which I would never do), some persistence and escalations meant that they finally agrees to stop targeting.

The result is that the website is for the community who wont need to compete against a naughty bot that doesn't add any value to anyone.


If this is all a bit cryptic, don't worry, it juts feels like something to be happy about.

Arbuckle23
Posts: 463
Joined: Sun Jan 11, 2015 7:07 pm
Location: Mornington Peninsula

Re: Bye Bye bing

Postby Arbuckle23 » Fri Apr 20, 2018 7:21 am

Well done

b4igo
Posts: 50
Joined: Sun Oct 12, 2008 4:15 pm
Location: Sydney

Re: Bye Bye bing

Postby b4igo » Fri Apr 20, 2018 7:32 am

Nice work! Appreciate all your effort to keep the site running smoothly for us all!!

User avatar
ValleyForge
Posts: 1607
Joined: Tue Jan 27, 2009 5:37 pm
Location: Brisbane

Re: Bye Bye bing

Postby ValleyForge » Fri Apr 20, 2018 9:50 pm

We need more bots to protect us from the bots.
Ha ha ha! Cookies on dowels.

User avatar
10speedsemiracer
Posts: 825
Joined: Sat Feb 11, 2017 7:38 pm
Location: Melbourne, Eastern Suburbs, near the hills,

Re: Bye Bye bing

Postby 10speedsemiracer » Fri Apr 20, 2018 10:21 pm

Well done Chris, many thanks.
Mmm, SunTour

User avatar
nickobec
Posts: 1858
Joined: Fri Mar 27, 2009 12:51 am
Location: Perth or 42km south as the singlespeed flies
Contact:

Re: Bye Bye bing

Postby nickobec » Sat Apr 21, 2018 11:59 am

Had a similar problem a couple years back, found out the hard way trying to slow bingbot down that if msnbot and bingbot find a rule specific to itself it will ignore all other rules:

So my robots.txt now start with
User-agent: msnbot
Disallow: /
User-agent: bingbot
Disallow: /

User avatar
MichaelB
Posts: 8819
Joined: Tue Feb 20, 2007 9:29 am
Location: Adelaide, South Australia

Re: Bye Bye bing

Postby MichaelB » Sat Apr 21, 2018 5:49 pm

You spoke to someone !!??????

User avatar
trailgumby
Posts: 13101
Joined: Sat Jan 03, 2009 5:30 pm
Location: Northern Beaches, Sydney
Contact:

Re: Bye Bye bing

Postby trailgumby » Sat Apr 21, 2018 6:32 pm

MichaelB wrote:You spoke to someone !!??????

Probably a robot.

User avatar
Thoglette
Posts: 3837
Joined: Thu Feb 19, 2009 1:01 pm

Re: Bye Bye bing

Postby Thoglette » Sat Apr 21, 2018 7:54 pm

Well done. MS should have better manners.
Stop handing them the stick! - Dave Moulton
"People are worthy of respect, ideas are not." Peter Ellerton, UQ

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 12982
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: Bye Bye bing

Postby AUbicycles » Mon Apr 23, 2018 4:07 pm

Bingbot was ignoring their specified crawl rates and disallows in robots.txt . Eventually I had to block their IPs but when they were still responsible for up to 85% of traffic when their waves started, even that is still a load.

User avatar
nickobec
Posts: 1858
Joined: Fri Mar 27, 2009 12:51 am
Location: Perth or 42km south as the singlespeed flies
Contact:

Re: Bye Bye bing

Postby nickobec » Mon Apr 23, 2018 4:22 pm

AUbicycles wrote:Bingbot was ignoring their specified crawl rates and disallows in robots.txt . Eventually I had to block their IPs but when they were still responsible for up to 85% of traffic when their waves started, even that is still a load.


My problem was trying to slow their crawl rates, with a bingbot specific line in robots.txt

Which meant bingbot actually obeyed that rule, but ignored all the other rules in robots.txt

Which meant bingbot would crash the server by accessing huge number of pages normally blocked by robots.txt

So I removed the bingbot specific lines and just blocked bingbot and msnbot with
User-agent: msnbot
Disallow: /
User-agent: bingbot
Disallow: /

Have not had any issues since, and that was years ago

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 12982
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: Bye Bye bing

Postby AUbicycles » Tue Apr 24, 2018 2:10 am

I went so far, in frustration, to sign up to their webmaster tools and follow their guidance and set up the site but even that didn’t improve.

Very specifically, I used their tools to force them to ignore specific directories and their error was (to the effect) “these pages are too valuable to bing so can’t be removed”

eeksll
Posts: 2451
Joined: Tue Oct 20, 2009 10:36 pm

Re: Bye Bye bing

Postby eeksll » Sun Apr 29, 2018 1:43 pm

just interested,

was this something they did on their end or did you also have to do something?

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 12982
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: Bye Bye bing

Postby AUbicycles » Mon Apr 30, 2018 2:39 am

At their end- they wanted me to open up the site and make it vulnerable to their flooding but I refused. Contact partner was fast in responding and polite.

Jmuzz
Posts: 344
Joined: Mon Jan 29, 2018 1:42 pm

Re: Bye Bye bing

Postby Jmuzz » Fri May 04, 2018 11:17 pm

Doesn't phpBB recognise all the bots as logged in users, and therefore let you assign them restricted permissions?
Or have they gone and defeated that by going stealth these days?

Shred11
Posts: 91
Joined: Sun May 18, 2014 12:32 pm
Location: Launceston

Re: Bye Bye bing

Postby Shred11 » Sat May 05, 2018 8:56 am

Does anyone actually use Bing anyway? Once in a blue moon, I resort to trying Bing and it just throws up all sorts of rubbish search results that don't help at all. duckduckgo.com isn't a bad alternative if Google isn't giving good results.

User avatar
10speedsemiracer
Posts: 825
Joined: Sat Feb 11, 2017 7:38 pm
Location: Melbourne, Eastern Suburbs, near the hills,

Re: Bye Bye bing

Postby 10speedsemiracer » Sat May 05, 2018 4:12 pm

I'm over Google. Duckduckgo is my preference.
Mmm, SunTour

User avatar
mikesbytes
Super Mod
Super Mod
Posts: 18825
Joined: Thu Dec 01, 2005 11:42 pm
Location: Tempe, Sydney
Contact:

Re: Bye Bye bing

Postby mikesbytes » Sat May 05, 2018 4:52 pm

Bing is popular in China, I'm sure there will be other demographic's where it has a client base
If the R-1 rule is broken, what happens to N+1?

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 12982
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: Bye Bye bing

Postby AUbicycles » Tue May 08, 2018 6:59 am

phpbb by default can do some recognition, but doesn't actively identify and control or block. The other things is that bots can spoof - while bing wasn't spoofing, it wasn't ethical either.

Does anyone use it? well microsoft forced it on people and for those who couldn't bypass, it was used. But it was never really very important even though microsoft tried to compete and, as always will push it as far as they can. As Mikebytes suggests, there are probably some demographics who rely on it but Google was always a step ahead for quality results.

User avatar
kb
Posts: 2423
Joined: Sun Nov 13, 2011 3:22 pm

Re: Bye Bye bing

Postby kb » Tue May 08, 2018 12:51 pm

I do like the use of quadtrees for mapping though.
Image

User avatar
mikesbytes
Super Mod
Super Mod
Posts: 18825
Joined: Thu Dec 01, 2005 11:42 pm
Location: Tempe, Sydney
Contact:

Re: Bye Bye bing

Postby mikesbytes » Wed May 09, 2018 8:26 pm

I suspect there are few searches done on Bing to find Australian bicycle info. A quick surf of the web has uncovered that 14% of searches in the US and Canada are done using Bing and that 85% of Bing's total searches are done by US users, however I think that statement has excluded China

https://www.aborg.com/2014/08/google-vs-bing-which-one-are-your-customers-using/

Most of the people using this site reside within Australia and the following page states that 98% of searches within Australia are thru Google, Yahoo and NineMSM. BUT what is underneath them? go to https://www.nine.com.au/?redirect=holdingpage and have a look at who's search it is.

BTW the remaining 2% includes specialty engines such as web wombat.
If the R-1 rule is broken, what happens to N+1?

Who is online

Users browsing this forum: No registered users