BNA Forum and Site Downtime
- AUbicycles
- Site Admin
- Posts: 15589
- Joined: Tue Aug 23, 2005 2:14 am
- Location: Sydney & Frankfurt
- Contact:
BNA Forum and Site Downtime
Postby AUbicycles » Mon Mar 19, 2018 6:39 pm
Unfortunately I was not aware of this earlier and couldn't respond.
It appears that BNA was (again) flooded by unethical bots. Their intention is to scrape the entire site as fast as possible which puts a massive load on the site and beyond preventing access, they also rapidly 'suck' all of the hosting credit so when they finally stop, then this continues to prevent genuine people visiting as the host sees that the entire 'contingent' has been used.
I will try and find out more about this to try and prevent - but thank you for the positive message, support and patience.
Christopher
- Derny Driver
- Posts: 3039
- Joined: Wed Nov 21, 2012 12:18 pm
- Location: Wollongong
Re: BNA Forum and Site Downtime
Postby Derny Driver » Mon Mar 19, 2018 6:52 pm
This place is great but we wont die if we cant get on for several hours.
We support you in the war against the bots
- MichaelB
- Posts: 14853
- Joined: Tue Feb 20, 2007 9:29 am
- Location: Adelaide, South Australia
Re: BNA Forum and Site Downtime
Postby MichaelB » Mon Mar 19, 2018 8:21 pm
The world didn’t end and the fact that people do this sort of stuff is saddening,but a frustrating part of life.
As usual, many thanks Christopher
- BianchiCam
- Posts: 667
- Joined: Mon Feb 18, 2013 12:42 pm
- Location: Sunny Coast. Oop Norf!
- sogood
- Posts: 17168
- Joined: Thu Aug 31, 2006 7:31 am
- Location: Sydney AU
Re: BNA Forum and Site Downtime
Postby sogood » Mon Mar 19, 2018 11:01 pm
RK wrote:And that is Wikipedia - I can write my own definition.
- 10speedsemiracer
- Posts: 4904
- Joined: Sat Feb 11, 2017 7:38 pm
- Location: Back on the Tools .. when I'm not in the office
Re: BNA Forum and Site Downtime
Postby 10speedsemiracer » Mon Mar 19, 2018 11:14 pm
We appreciate your hard work btw.
- MichaelB
- Posts: 14853
- Joined: Tue Feb 20, 2007 9:29 am
- Location: Adelaide, South Australia
Re: BNA Forum and Site Downtime
Postby MichaelB » Tue Mar 20, 2018 7:58 am
- AUbicycles
- Site Admin
- Posts: 15589
- Joined: Tue Aug 23, 2005 2:14 am
- Location: Sydney & Frankfurt
- Contact:
Re: BNA Forum and Site Downtime
Postby AUbicycles » Tue Mar 20, 2018 10:02 am
Sometime there is brute force hacking to try and break in.
Sometimes it is dubious ‘services’ that crawl the site bug do it poorly which affects access and don’t use thd content they gain in any useful way (ie dodgy, unethical or scam).
I have some tools in place to limit the baddies but the bad ones utilise bot nets and attack from many different Ips.
When I consider spam rnails and the non-genuine traffic, it would be fair to say that at least half the traffic on the entire is not genuine.
- g-boaf
- Posts: 21457
- Joined: Mon Sep 26, 2011 6:11 pm
Re: BNA Forum and Site Downtime
Postby g-boaf » Tue Mar 20, 2018 7:52 pm
It's a bloody annoying battle to fight against them. Never ends either. Kudos for trying to keep on top of it all.AUbicycles wrote:Thank you for the email and facebook reports.
Unfortunately I was not aware of this earlier and couldn't respond.
It appears that BNA was (again) flooded by unethical bots. Their intention is to scrape the entire site as fast as possible which puts a massive load on the site and beyond preventing access, they also rapidly 'suck' all of the hosting credit so when they finally stop, then this continues to prevent genuine people visiting as the host sees that the entire 'contingent' has been used.
I will try and find out more about this to try and prevent - but thank you for the positive message, support and patience.
Christopher
- AUbicycles
- Site Admin
- Posts: 15589
- Joined: Tue Aug 23, 2005 2:14 am
- Location: Sydney & Frankfurt
- Contact:
Re: BNA Forum and Site Downtime
Postby AUbicycles » Wed Mar 21, 2018 6:02 pm
In short - it has the effect of a DOS - Denial of Service attack although I can't say it it targeted, rather it is more likely simply aggressive (and illegal, unethical etc).
- AUbicycles
- Site Admin
- Posts: 15589
- Joined: Tue Aug 23, 2005 2:14 am
- Location: Sydney & Frankfurt
- Contact:
Re: BNA Forum and Site Downtime
Postby AUbicycles » Wed Mar 21, 2018 7:14 pm
- kb
- Posts: 2570
- Joined: Sun Nov 13, 2011 3:22 pm
Re: BNA Forum and Site Downtime
Postby kb » Thu Mar 22, 2018 1:49 pm
- 10speedsemiracer
- Posts: 4904
- Joined: Sat Feb 11, 2017 7:38 pm
- Location: Back on the Tools .. when I'm not in the office
Re: BNA Forum and Site Downtime
Postby 10speedsemiracer » Thu Mar 22, 2018 2:01 pm
Be honest, the cleaner unplugged the power cable and the UPS wasn't plugged in...(only joking)AUbicycles wrote:Another small hiccup - but my doing this time as I am adjusting a SSL setting.
-
- Posts: 657
- Joined: Wed Oct 28, 2009 8:26 pm
Re: BNA Forum and Site Downtime
Postby Mike Ayling » Fri Mar 23, 2018 8:34 am
Neil who can get a bit paranoid at times immediately blamed a former site member.
Are you aware of any other cycling related sites which have been attacked?
Mike
- im_no_pro
- Super Mod
- Posts: 6029
- Joined: Thu Oct 23, 2008 10:29 pm
- Location: Geelong
Re: BNA Forum and Site Downtime
Postby im_no_pro » Fri Mar 23, 2018 1:37 pm
I was going to text you but I lost track of what country you are in and whether the number I have even still exists, so FB message it wasAUbicycles wrote:Thank you for the email and facebook reports.
Unfortunately I was not aware of this earlier and couldn't respond.
master6 wrote: Moderators are like Club Handicappers; I often think they are wrong, but I dont want the job.
- AUbicycles
- Site Admin
- Posts: 15589
- Joined: Tue Aug 23, 2005 2:14 am
- Location: Sydney & Frankfurt
- Contact:
Re: BNA Forum and Site Downtime
Postby AUbicycles » Fri Mar 23, 2018 6:53 pm
In the meantime I may be able to configure it a bit - but as I am stuck without access to original log files - it is a bit tough. At the moment it appears to be a forum issue - so it is the forum rather than the main site that it being overrun.
As I am travelling - I have limited time but will try and keep an eye on it and at least attempt to get it back online as soon as I can.
- AUbicycles
- Site Admin
- Posts: 15589
- Joined: Tue Aug 23, 2005 2:14 am
- Location: Sydney & Frankfurt
- Contact:
Re: BNA Forum and Site Downtime
Postby AUbicycles » Tue Mar 27, 2018 7:47 am
They were flooding the site with a massive amount of requests - this put our CPU load to maximum, used all of the CPU credit and knocked the site offline until I could manually replenish the credit. The effect of a DOS attack (Denial of Service) which also lasted longer because all the credit was gone.
Unfortunately, it is not easy to stop but I have taken some measures to try... and hope that they stop it themselves as it is a very destructive bot and is essentially unethical because it floods and requests the same file thousands of times a second. Their website lists some fairly reputable clients such as Emirates, Canon, Nestlé and Mont Blanc so it is unfortunate that with what appears to be an impressive client-list, they need to resort to such unethical activity.
Here is small sample for the techies.
Also, Microsoft Bing is just as bad. While the load is not as heavy, they are permanently crawling the website and the bingbot ignores the rules - these are rules which Microsoft / BingBot confirms on their website - but the delay and even a disallow has no effect and bingbot has a fairly big load - up to 80% of the traffic (load) at times.
Cheers
Christopher
209.123.42.49[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.44[26/Mar/2018:18:38:32 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.45[26/Mar/2018:18:38:35 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.41.250[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.41.253[26/Mar/2018:18:38:36 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.52[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.44[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.53[26/Mar/2018:18:38:31 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.44[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.48[26/Mar/2018:18:38:31 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:32 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot http://www.customscoop.com"
207.99.127.20[26/Mar/2018:18:38:35 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.61[26/Mar/2018:18:38:35 +0000] "GET / HTTP/1.1" 200 54924 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.52[26/Mar/2018:18:38:31 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.62[26/Mar/2018:18:38:36 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.50[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54924 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.41.252[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.54[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:37 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:32 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.41.252[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:37 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.54[26/Mar/2018:18:38:37 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.50[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
207.99.127.20[26/Mar/2018:18:38:37 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot http://www.customscoop.com"
209.123.42.44[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.40[26/Mar/2018:18:38:34 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
209.123.42.54[26/Mar/2018:18:38:37 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
- MichaelB
- Posts: 14853
- Joined: Tue Feb 20, 2007 9:29 am
- Location: Adelaide, South Australia
Re: BNA Forum and Site Downtime
Postby MichaelB » Tue Mar 27, 2018 8:45 am
Glad you know what's going on !!
- 10speedsemiracer
- Posts: 4904
- Joined: Sat Feb 11, 2017 7:38 pm
- Location: Back on the Tools .. when I'm not in the office
Re: BNA Forum and Site Downtime
Postby 10speedsemiracer » Tue Mar 27, 2018 5:00 pm
Now knowing the IP address of the main culprit, can you not block the bot using something like .htaccess ? Or are they aggressive enough to get around this ?AUbicycles wrote:Update - I finally found the culprit - a 'service' called Carma who were flooding BNA with their 'ScooperBot'.
They were flooding the site with a massive amount of requests - ......
209.123.42.49[26/Mar/2018:18:38:33 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
...
209.123.42.54[26/Mar/2018:18:38:37 +0000] "GET / HTTP/1.1" 200 54925 "-" "ScooperBot/3.0 (+http://www.carma.com)"
- AUbicycles
- Site Admin
- Posts: 15589
- Joined: Tue Aug 23, 2005 2:14 am
- Location: Sydney & Frankfurt
- Contact:
Re: BNA Forum and Site Downtime
Postby AUbicycles » Tue Mar 27, 2018 5:37 pm
A person from Carma did get back and they said they will remove BNA so it just leaves the nasty bingbot.
- 10speedsemiracer
- Posts: 4904
- Joined: Sat Feb 11, 2017 7:38 pm
- Location: Back on the Tools .. when I'm not in the office
Re: BNA Forum and Site Downtime
Postby 10speedsemiracer » Tue Mar 27, 2018 6:06 pm
- AUbicycles
- Site Admin
- Posts: 15589
- Joined: Tue Aug 23, 2005 2:14 am
- Location: Sydney & Frankfurt
- Contact:
Re: BNA Forum and Site Downtime
Postby AUbicycles » Tue Mar 27, 2018 7:36 pm
- 10speedsemiracer
- Posts: 4904
- Joined: Sat Feb 11, 2017 7:38 pm
- Location: Back on the Tools .. when I'm not in the office
Re: BNA Forum and Site Downtime
Postby 10speedsemiracer » Tue Mar 27, 2018 7:50 pm
Which is why Bing is the #1 choice when it comes to search engines....oh waitAUbicycles wrote:Correct... I tried and the chat robot said "would you like to speak to a person?" I chose Yes. Then was provided a link to the bing community discussion forum... what a joke.
Return to “Info / announcements”
- General Australian Cycling Topics
- Info / announcements
- Buying a bike / parts
- General Cycling Discussion
- The Bike Shed
- Cycling Health
- Cycling Safety and Advocacy
- Women's Cycling
- Bike & Gear Reviews
- Cycling Trade
- Stolen Bikes
- Bicycle FAQs
- The Market Place
- Member to Member Bike and Gear Sales
- Want to Buy, Group Buy, Swap
- My Bikes or Gear Elsewhere
- Serious Biking
- Audax / Randonneuring
- Retro biking
- Commuting
- MTB
- Recumbents
- Fixed Gear/ Single Speed
- Track
- Electric Bicycles
- Cyclocross and Gravel Grinding
- Dragsters / Lowriders / Cruisers
- Children's Bikes
- Cargo Bikes and Utility Cycling
- Road Racing
- Road Biking
- Training
- Time Trial
- Triathlon
- International and National Tours and Events
- Cycle Touring
- Touring Australia
- Touring Overseas
- Touring Bikes and Equipment
- Australia
- Western Australia
- New South Wales
- Queensland
- South Australia
- Victoria
- ACT
- Tasmania
- Northern Territory
- Country & Regional
Who is online
Users browsing this forum: No registered users
- All times are UTC+10:00
- Top
- Delete cookies
About the Australian Cycling Forums
The Australian Cycling Forums is a welcoming community where you can ask questions and talk about the type of bikes and cycling topics you like.
Bicycles Network Australia
Forum Information
Connect with BNA
This website uses affiliate links to retail platforms including ebay, amazon, proviz and ribble.