登录注册
Quick Links : Mindat手册The Rock H. Currier Digital LibraryMindat Newsletter [Free Download]
主页关于 MindatMindat手册Mindat的历史版权Who We Are联系我们于 Mindat.org刊登广告
捐赠给 MindatCorporate Sponsorship赞助板页已赞助的板页在 Mindat刊登 广告的广告商于 Mindat.org刊登广告
Learning CenterWhat is a mineral?The most common minerals on earthInformation for EducatorsMindat ArticlesThe ElementsThe Rock H. Currier Digital LibraryGeologic Time
搜索矿物的性质搜索矿物的化学Advanced Locality Search随意显示任何一 种矿物Random Locality使用minID搜索邻近产地Search Articles搜索词汇表更多搜索选项
搜索:
矿物名称:
地区产地名称:
关键字:
 
Mindat手册添加新照片Rate Photos产区编辑报告Coordinate Completion Report添加词汇表项目
Mining Companies统计会员列表Mineral MuseumsClubs & Organizations矿物展及活动The Mindat目录表设备设置The Mineral Quiz
照片搜索Photo GalleriesSearch by Color今天最新的照片昨天最新的照片用户照片相集过去每日精选照片相集Photography

Improving Mindat.orgAPI returns 403 from different IPs

16th Apr 2024 12:54 UTCSam Leeflang

Dear Mindat,

We sometimes use your locality endpoint to add coordinates to our specimen information.
We made a small service which collects information from the specimen, calls the Mindat API and adds the possible result to the specimen information.
The tool is in piloting phase, so we do a couple each month and maybe a bit more when we test, but we try to keep the calls to a minimum.
However, we noticed that when we run the tool on the server, we get a 403 Forbidden response.
Locally, from which we most calls, it works fine, so I don't expect it to be rate related.
I also tested this from another server, from which we have never done any calls, and it also returned a 403, however a third server gave back a 200.
We were wondering if there is any explanation why some servers give us a positive response while other give a 403 all with the same request and the same token?
The servers with which we tested were either our local machines or AWS servers which we run from in eu-west-2.
Any help or explanation will be appreciated.

Kind regards,
Sam

16th Apr 2024 13:56 UTCDavid Von Bargen Manager

In general, when you get a 403 forbidden, it is because a mindat user has been naughty  (trying to download the entire site), and the IP address (or someone with a nearby IP address - and used multiple close addresses - where we flag a range of addresses) has been flagged to not allow access - this activity has severe repercussions for the rest of the people using mindat.  

16th Apr 2024 15:13 UTCSam Leeflang

Hi David,

Thanks for your quick response!
I understand that you blacklist IP address and ranges.
However, we tried with multiple servers, all with different IPs, for some we are certain we never made a call before the single call today.
Is it possible that large parts of the AWS IP ranges have been blacklisted?

To be more precise, we tried from:
- 35.156.85.74/32 -> eu-central-1 : First time making a call from this machine, received a 403
- 3.10.241.217/32 -> eu-west-2 : First time making a call from this machine, received a 200  
- 18.130.232.162/32 -> eu-west-2 : First time making a call from this machine, received a 403 
- 18.134.153.167/32 -> eu-west-2 : There might have been a couple calls in the past six months, received a 403 

Kind regards,
Sam

16th Apr 2024 15:49 UTCJolyon Ralph Founder

Yes, large parts of AWS have indeed been blacklisted - mostly because AWS is frequently used by bad crawler scripts (eg for training AIs on mindat content without permission).

You’ll need a fixed IP and then we can whitelist it - or don’t use AWS.

17th Apr 2024 08:35 UTCSam Leeflang

Jolyon Ralph Founder  ✉️

Yes, large parts of AWS have indeed been blacklisted - mostly because AWS is frequently used by bad crawler scripts (eg for training AIs on mindat content without permission).

You’ll need a fixed IP and then we can whitelist it - or don’t use AWS.
Thanks Jolyon for this explanation.
We run on a Kubernetes cluster with auto-scaling resources, making a fixed IP a bit tricky.
I have put fixing the egress IP on our backlog, and we will get back to you once we have this done.
It might take some time as it is not our top priority at the moment.

Kind regards,
Sam
 
and/or  
Mindat Discussions Facebook Logo Instagram Logo Discord Logo
版权所有© mindat.org1993年至2024年,除了规定的地方。 Mindat.org全赖于全球数千个以上成员和支持者们的参与。
隐私政策 - 条款和条款细则 - 联络我们 - Report a bug/vulnerability Current server date and time: 2024.5.3 02:58:51
Go to top of page