Learning CenterWhat is a mineral?The most common minerals on earthInformation for EducatorsMindat ArticlesThe ElementsThe Rock H. Currier Digital LibraryGeologic Time
搜索矿物的性质搜索矿物的化学Advanced Locality Search随意显示任何一 种矿物Random Locality使用minID搜索邻近产地Search Articles搜索词汇表更多搜索选项
╳Discussions
💬 Home🔎 Search📅 LatestGroups
EducationOpen discussion area.Fakes & FraudsOpen discussion area.Field CollectingOpen discussion area.FossilsOpen discussion area.Gems and GemologyOpen discussion area.GeneralOpen discussion area.How to ContributeOpen discussion area.Identity HelpOpen discussion area.Improving Mindat.orgOpen discussion area.LocalitiesOpen discussion area.Lost and Stolen SpecimensOpen discussion area.MarketplaceOpen discussion area.MeteoritesOpen discussion area.Mindat ProductsOpen discussion area.Mineral ExchangesOpen discussion area.Mineral PhotographyOpen discussion area.Mineral ShowsOpen discussion area.Mineralogical ClassificationOpen discussion area.Mineralogy CourseOpen discussion area.MineralsOpen discussion area.Minerals and MuseumsOpen discussion area.PhotosOpen discussion area.Techniques for CollectorsOpen discussion area.The Rock H. Currier Digital LibraryOpen discussion area.UV MineralsOpen discussion area.Recent Images in Discussions
Improving Mindat.orgAPI returns 403 from different IPs
16th Apr 2024 12:54 UTCSam Leeflang
We sometimes use your locality endpoint to add coordinates to our specimen information.
We made a small service which collects information from the specimen, calls the Mindat API and adds the possible result to the specimen information.
The tool is in piloting phase, so we do a couple each month and maybe a bit more when we test, but we try to keep the calls to a minimum.
However, we noticed that when we run the tool on the server, we get a 403 Forbidden response.
Locally, from which we most calls, it works fine, so I don't expect it to be rate related.
Locally, from which we most calls, it works fine, so I don't expect it to be rate related.
I also tested this from another server, from which we have never done any calls, and it also returned a 403, however a third server gave back a 200.
We were wondering if there is any explanation why some servers give us a positive response while other give a 403 all with the same request and the same token?
The servers with which we tested were either our local machines or AWS servers which we run from in eu-west-2.
The servers with which we tested were either our local machines or AWS servers which we run from in eu-west-2.
Any help or explanation will be appreciated.
Kind regards,
Sam
16th Apr 2024 13:56 UTCDavid Von Bargen Manager
In general, when you get a 403 forbidden, it is because a mindat user has been naughty (trying to download the entire site), and the IP address (or someone with a nearby IP address - and used multiple close addresses - where we flag a range of addresses) has been flagged to not allow access - this activity has severe repercussions for the rest of the people using mindat.
16th Apr 2024 15:13 UTCSam Leeflang
Thanks for your quick response!
I understand that you blacklist IP address and ranges.
However, we tried with multiple servers, all with different IPs, for some we are certain we never made a call before the single call today.
Is it possible that large parts of the AWS IP ranges have been blacklisted?
To be more precise, we tried from:
- 35.156.85.74/32 -> eu-central-1 : First time making a call from this machine, received a 403
- 3.10.241.217/32 -> eu-west-2 : First time making a call from this machine, received a 200
- 18.130.232.162/32 -> eu-west-2 : First time making a call from this machine, received a 403
- 18.134.153.167/32 -> eu-west-2 : There might have been a couple calls in the past six months, received a 403
Kind regards,
Sam
Sam
16th Apr 2024 15:49 UTCJolyon Ralph Founder
You’ll need a fixed IP and then we can whitelist it - or don’t use AWS.
17th Apr 2024 08:35 UTCSam Leeflang
Jolyon Ralph Founder ✉️
Yes, large parts of AWS have indeed been blacklisted - mostly because AWS is frequently used by bad crawler scripts (eg for training AIs on mindat content without permission).You’ll need a fixed IP and then we can whitelist it - or don’t use AWS.
Thanks Jolyon for this explanation.
We run on a Kubernetes cluster with auto-scaling resources, making a fixed IP a bit tricky.
I have put fixing the egress IP on our backlog, and we will get back to you once we have this done.
It might take some time as it is not our top priority at the moment.
Kind regards,
Sam
版权所有© mindat.org1993年至2024年,除了规定的地方。 Mindat.org全赖于全球数千个以上成员和支持者们的参与。
隐私政策 - 条款和条款细则 - 联络我们 - Report a bug/vulnerability Current server date and time: 2024.5.3 02:58:51
隐私政策 - 条款和条款细则 - 联络我们 - Report a bug/vulnerability Current server date and time: 2024.5.3 02:58:51