BNA Forum and Site Downtime

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15583
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

BNA Forum and Site Downtime

Postby AUbicycles » Mon Mar 19, 2018 6:39 pm

Thank you for the email and facebook reports.
Unfortunately I was not aware of this earlier and couldn't respond.

It appears that BNA was (again) flooded by unethical bots. Their intention is to scrape the entire site as fast as possible which puts a massive load on the site and beyond preventing access, they also rapidly 'suck' all of the hosting credit so when they finally stop, then this continues to prevent genuine people visiting as the host sees that the entire 'contingent' has been used.

I will try and find out more about this to try and prevent - but thank you for the positive message, support and patience.


Christopher
Cycling is in my BNA

User avatar
Derny Driver
Posts: 3039
Joined: Wed Nov 21, 2012 12:18 pm
Location: Wollongong

Re: BNA Forum and Site Downtime

Postby Derny Driver » Mon Mar 19, 2018 6:52 pm

Not your fault Chris.
This place is great but we wont die if we cant get on for several hours.
We support you in the war against the bots

User avatar
MichaelB
Posts: 14775
Joined: Tue Feb 20, 2007 9:29 am
Location: Adelaide, South Australia

Re: BNA Forum and Site Downtime

Postby MichaelB » Mon Mar 19, 2018 8:21 pm

+1 to what DD said.

The world didn’t end and the fact that people do this sort of stuff is saddening,but a frustrating part of life.

As usual, many thanks Christopher :-)

User avatar
BianchiCam
Posts: 667
Joined: Mon Feb 18, 2013 12:42 pm
Location: Sunny Coast. Oop Norf!

Re: BNA Forum and Site Downtime

Postby BianchiCam » Mon Mar 19, 2018 8:58 pm

Friggen hate Bots

User avatar
sogood
Posts: 17168
Joined: Thu Aug 31, 2006 7:31 am
Location: Sydney AU

Re: BNA Forum and Site Downtime

Postby sogood » Mon Mar 19, 2018 11:01 pm

What's attracting those bots to this site? What value do they gain?
Bianchi, Ridley, Tern, Montague and All things Apple :)
RK wrote:And that is Wikipedia - I can write my own definition.

User avatar
10speedsemiracer
Posts: 4904
Joined: Sat Feb 11, 2017 7:38 pm
Location: Back on the Tools .. when I'm not in the office

Re: BNA Forum and Site Downtime

Postby 10speedsemiracer » Mon Mar 19, 2018 11:14 pm

Stupid bloody search engines...can you identify the bots via IP or name and block them at Server ?

We appreciate your hard work btw.
Campagnolo for show, SunTour for go

User avatar
MichaelB
Posts: 14775
Joined: Tue Feb 20, 2007 9:29 am
Location: Adelaide, South Australia

Re: BNA Forum and Site Downtime

Postby MichaelB » Tue Mar 20, 2018 7:58 am

WW forum has also had problems in logging in. So no idea if it’s the same issue or something else

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15583
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: BNA Forum and Site Downtime

Postby AUbicycles » Tue Mar 20, 2018 10:02 am

Sometimes they are scraping the site, then duplicate the site with manipulated content to try and trick genuine search engines.

Sometime there is brute force hacking to try and break in.

Sometimes it is dubious ‘services’ that crawl the site bug do it poorly which affects access and don’t use thd content they gain in any useful way (ie dodgy, unethical or scam).

I have some tools in place to limit the baddies but the bad ones utilise bot nets and attack from many different Ips.

When I consider spam rnails and the non-genuine traffic, it would be fair to say that at least half the traffic on the entire is not genuine.
Cycling is in my BNA

User avatar
g-boaf
Posts: 21320
Joined: Mon Sep 26, 2011 6:11 pm

Re: BNA Forum and Site Downtime

Postby g-boaf » Tue Mar 20, 2018 7:52 pm

AUbicycles wrote:Thank you for the email and facebook reports.
Unfortunately I was not aware of this earlier and couldn't respond.

It appears that BNA was (again) flooded by unethical bots. Their intention is to scrape the entire site as fast as possible which puts a massive load on the site and beyond preventing access, they also rapidly 'suck' all of the hosting credit so when they finally stop, then this continues to prevent genuine people visiting as the host sees that the entire 'contingent' has been used.

I will try and find out more about this to try and prevent - but thank you for the positive message, support and patience.


Christopher
It's a bloody annoying battle to fight against them. Never ends either. Kudos for trying to keep on top of it all.

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15583
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: BNA Forum and Site Downtime

Postby AUbicycles » Wed Mar 21, 2018 6:02 pm

Been hit again. Unfortunately I can get log details and have to try and work out how I can get the crucial info on who is flooding BNA.

In short - it has the effect of a DOS - Denial of Service attack although I can't say it it targeted, rather it is more likely simply aggressive (and illegal, unethical etc).
Cycling is in my BNA

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15583
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: BNA Forum and Site Downtime

Postby AUbicycles » Wed Mar 21, 2018 7:14 pm

Another small hiccup - but my doing this time as I am adjusting a SSL setting.
Cycling is in my BNA

User avatar
kb
Posts: 2570
Joined: Sun Nov 13, 2011 3:22 pm

Re: BNA Forum and Site Downtime

Postby kb » Thu Mar 22, 2018 1:49 pm

Bruce Schneier had some interesting musings in his latest Cryptogram newsletter. He postulated that machine learning could tip the threat/response balance in cyber security more towards the defence side than it currently is. Not that that helps now...
Image

User avatar
10speedsemiracer
Posts: 4904
Joined: Sat Feb 11, 2017 7:38 pm
Location: Back on the Tools .. when I'm not in the office

Re: BNA Forum and Site Downtime

Postby 10speedsemiracer » Thu Mar 22, 2018 2:01 pm

AUbicycles wrote:Another small hiccup - but my doing this time as I am adjusting a SSL setting.
Be honest, the cleaner unplugged the power cable and the UPS wasn't plugged in...(only joking)
Campagnolo for show, SunTour for go

Mike Ayling
Posts: 657
Joined: Wed Oct 28, 2009 8:26 pm

Re: BNA Forum and Site Downtime

Postby Mike Ayling » Fri Mar 23, 2018 8:34 am

The CrazyGuyonabike site reported a similar attack a little while back.

Neil who can get a bit paranoid at times immediately blamed a former site member.

Are you aware of any other cycling related sites which have been attacked?

Mike
Recreational e bikes - for the sick, lame and lazy!

User avatar
im_no_pro
Super Mod
Super Mod
Posts: 6029
Joined: Thu Oct 23, 2008 10:29 pm
Location: Geelong

Re: BNA Forum and Site Downtime

Postby im_no_pro » Fri Mar 23, 2018 1:37 pm

AUbicycles wrote:Thank you for the email and facebook reports.
Unfortunately I was not aware of this earlier and couldn't respond.
I was going to text you but I lost track of what country you are in and whether the number I have even still exists, so FB message it was :)
master6 wrote: Moderators are like Club Handicappers; I often think they are wrong, but I dont want the job.

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15583
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: BNA Forum and Site Downtime

Postby AUbicycles » Fri Mar 23, 2018 6:53 pm

It seems to be a pattern now - has happened again and I think I got it fairly swiftly.

In the meantime I may be able to configure it a bit - but as I am stuck without access to original log files - it is a bit tough. At the moment it appears to be a forum issue - so it is the forum rather than the main site that it being overrun.

As I am travelling - I have limited time but will try and keep an eye on it and at least attempt to get it back online as soon as I can.
Cycling is in my BNA

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15583
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: BNA Forum and Site Downtime

Postby AUbicycles » Tue Mar 27, 2018 7:47 am

Update - I finally found the culprit - a 'service' called Carma who were flooding BNA with their 'ScooperBot'.

They were flooding the site with a massive amount of requests - this put our CPU load to maximum, used all of the CPU credit and knocked the site offline until I could manually replenish the credit. The effect of a DOS attack (Denial of Service) which also lasted longer because all the credit was gone.

Unfortunately, it is not easy to stop but I have taken some measures to try... and hope that they stop it themselves as it is a very destructive bot and is essentially unethical because it floods and requests the same file thousands of times a second. Their website lists some fairly reputable clients such as Emirates, Canon, Nestlé and Mont Blanc so it is unfortunate that with what appears to be an impressive client-list, they need to resort to such unethical activity.

Here is small sample for the techies.

Also, Microsoft Bing is just as bad. While the load is not as heavy, they are permanently crawling the website and the bingbot ignores the rules - these are rules which Microsoft / BingBot confirms on their website - but the delay and even a disallow has no effect and bingbot has a fairly big load - up to 80% of the traffic (load) at times.

Cheers
Christopher


209.123.42.49[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.44[26/Mar/2018:18:38:32 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.45[26/Mar/2018:18:38:35 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.41.250[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.41.253[26/Mar/2018:18:38:36 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.52[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.44[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.53[26/Mar/2018:18:38:31 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.44[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.48[26/Mar/2018:18:38:31 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:32 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot http://www.customscoop.com"
207.99.127.20[26/Mar/2018:18:38:35 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.61[26/Mar/2018:18:38:35 +0000] "GET / HTTP/1.1" 200 54924 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.52[26/Mar/2018:18:38:31 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.62[26/Mar/2018:18:38:36 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.50[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54924 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.41.252[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.54[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:37 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:32 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.41.252[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:37 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.54[26/Mar/2018:18:38:37 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.50[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:37 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot http://www.customscoop.com"
209.123.42.44[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.40[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.54[26/Mar/2018:18:38:37 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
Cycling is in my BNA

User avatar
MichaelB
Posts: 14775
Joined: Tue Feb 20, 2007 9:29 am
Location: Adelaide, South Australia

Re: BNA Forum and Site Downtime

Postby MichaelB » Tue Mar 27, 2018 8:45 am

Keep up the GREAT work Christopher. I'll amdit that 50% of the stuff you said above wizzes past me like an unrepentant doper on a ride.

Glad you know what's going on !!

User avatar
10speedsemiracer
Posts: 4904
Joined: Sat Feb 11, 2017 7:38 pm
Location: Back on the Tools .. when I'm not in the office

Re: BNA Forum and Site Downtime

Postby 10speedsemiracer » Tue Mar 27, 2018 5:00 pm

AUbicycles wrote:Update - I finally found the culprit - a 'service' called Carma who were flooding BNA with their 'ScooperBot'.

They were flooding the site with a massive amount of requests - ......


209.123.42.49[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
...

209.123.42.54[26/Mar/2018:18:38:37 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
Now knowing the IP address of the main culprit, can you not block the bot using something like .htaccess ? Or are they aggressive enough to get around this ?
Campagnolo for show, SunTour for go

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15583
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: BNA Forum and Site Downtime

Postby AUbicycles » Tue Mar 27, 2018 5:37 pm

Yes - as soon as I found it out I put in the block they can still flood the site. I am tracking surges where up to 80% of all traffic is getting blocked.

A person from Carma did get back and they said they will remove BNA so it just leaves the nasty bingbot.
Cycling is in my BNA

User avatar
10speedsemiracer
Posts: 4904
Joined: Sat Feb 11, 2017 7:38 pm
Location: Back on the Tools .. when I'm not in the office

Re: BNA Forum and Site Downtime

Postby 10speedsemiracer » Tue Mar 27, 2018 6:06 pm

I doubt Bing will be as accommodating as Carma have apparently been...
Campagnolo for show, SunTour for go

User avatar
AUbicycles
Site Admin
Site Admin
Posts: 15583
Joined: Tue Aug 23, 2005 2:14 am
Location: Sydney & Frankfurt
Contact:

Re: BNA Forum and Site Downtime

Postby AUbicycles » Tue Mar 27, 2018 7:36 pm

Correct... I tried and the chat robot said "would you like to speak to a person?" I chose Yes. Then was provided a link to the bing community discussion forum... what a joke.
Cycling is in my BNA

User avatar
10speedsemiracer
Posts: 4904
Joined: Sat Feb 11, 2017 7:38 pm
Location: Back on the Tools .. when I'm not in the office

Re: BNA Forum and Site Downtime

Postby 10speedsemiracer » Tue Mar 27, 2018 7:50 pm

AUbicycles wrote:Correct... I tried and the chat robot said "would you like to speak to a person?" I chose Yes. Then was provided a link to the bing community discussion forum... what a joke.
Which is why Bing is the #1 choice when it comes to search engines....oh wait
Campagnolo for show, SunTour for go

Who is online

Users browsing this forum: blizzard, find_bruce