Latest version
Close
-->
Released:
A module to parse metadata out of urls and html documents
Project description
- MetaImage 1.6.0 Multilingual macOS 26 mb MetaImage is the ultimate tool to read, write and edit images metadata. This is first editor that allows you to edit EXIF, IPTC and XMP tags as well as MakerNotes in a beautiful and familiar interface.
- The initial-scale=1.0 part sets the initial zoom level when the page is first loaded by the browser. Here is an example of a web page without the viewport meta tag, and the same web page with the viewport meta.
Create a customized Pie Chart for free. Enter any data, customize the chart's colors, fonts and other details, then download it or easily share it with a shortened url Meta-Chart.com!
MetadataParser
Build Status: ![Python package](https://github.com/jvanasco/metadata_parser/workflows/Python%20package/badge.svg)
Meta Image 1 6 13 Commentary
Adobe incopy cc 2020 v15 0 1. MetadataParser is a Python module for pulling metadata out of web documents.
It requires BeautifulSoup , and was largely based on Erik River’s opengraph module ( https://github.com/erikriver/opengraph ).
I needed something more aggressive than Erik’s module, so had to fork.
Installation Recommendation
I strongly suggest you use the requests library version 2.4.3 or newer
This is not required, but it is better. On earlier versions it is possible to have an uncaught DecodeError exception when there is an underlying redirect/404. Recent fixes to requests improve redirect handling, urllib3 and urllib3 errors.
Features
- it pulls as much metadata out of a document as possible
- you can set a ‘strategy’ for finding metadata (i.e. only accept opengraph or page attributes)
- lightweight but functional(!) url validation
- logging is verbose, but nested under __debug__ statements, so it is compiled away when PYTHONOPTIMIZE is set
Notes
- This requires BeautifulSoup 4.
- For speed, it will instantiate a BeautifulSoup parser with lxml , and fall back to ‘none’ (the internal pure Python) if it can’t load lxml
- URL Validation is not RFC compliant, but tries to be “Real World” compliant
- It is HIGHLY recommended that you install lxml for usage. It is considerably faster. Considerably faster. *
You should also use a very recent version of lxml. I’ve had problems with segfaults on some versions < 2.3.x ; i would suggest using the most recent 3.x if possible.
The default ‘strategy’ is to look in this order:
Which stands for the following:
You can specify a strategy as a comma-separated list of the above.
The only 2 page elements currently supported are:
‘metadata’ elements are supported by name and property.
The MetadataParser object also wraps some convenience functions , which can be used otherwise , that are designed to turn alleged urls into well formed urls.
For example, you may pull a page:
Meta Image 1 6 13 Reasons
and that file indicates a canonical url which is simple “/file.html”.
This package will try to ‘remount’ the canonical url to the absolute url of “http://www.example.com/file.html” . It will return None if the end result is not a valid url.
This all happens under-the-hood, and is honestly really useful when dealing with indexers and spiders.
URL Validation
“Real World” URL validation is enabled by default. This is not RFC compliant.
There are a few gaps in the RFCs that allow for “odd behavior”. Just about any use-case for this package will desire/expect rules that parse URLs “in the wild”, not theoretical.
The differences:
- If an entirely numeric ip address is encountered, it is assumed to be a dot-notation IPV4 and it is checked to have the right amount of valid octets.The default behavior is to invalidate these hosts:According to RFCs those are valid hostnames that would fail as “IP Addresses” but pass as “Domain Names”. However in the real world, one would never encounter domain names like those.
- The only non-domain hostname that is allowed, is “localhost”The default behavior is to invalidate these hostsThose are considered to be valid hosts, and might exist on a local network or custom hosts file. However, they are not part of the public internet.
Although this behavior breaks RFCs, it greatly reduces the number of “False Positives” generated when analyzing internet pages. If you want to include bad data, you can submit a kwarg to MetadataParser.__init__
Handling Bad URLs and Encoded URIs
This library tries to safeguard against a few common situations.
# Encoded URIs and relative urls
Most website publishers will define an image as a URL:
Some will define an image as an encoded URI:
By default, the get_metadata_link() method can be used to ensure a valid link is extracted from the metadata payload:
This method accepts a kwarg allow_encoded_uri (default False) which will return the image without further processing:
Similarly, if a url is local:
The get_metadata_link method will automatically upgrade it onto the domain:
# Poorly Constructed Canonical URLs
Many website publishers implement canonical URLs incorrectly. This package tries to fix that.
By default MetadataParser is constructed with require_public_netloc=True and allow_localhosts=True.
This will require somewhat valid ‘public’ network locations in the url.
For example, these will all be valid URLs:
If these known ‘localhost’ urls are not wanted, they can be filtered out with allow_localhosts=False:
There are two convenience methods that can be used to get a canonical url or calculate the effective url:
These both accept an argument require_public_global, which defaults to True.
Assuming we have the following content on the url http://example.com/path/to/foo:
By default, versions 0.9.0 and later will detect ‘localhost:8000’ as an improper canonical url, and remount the local part “/alt-path/to/foo” onto the domain that served the file. The vast majority of times this ‘behavior’ has been encountered, this is the intended canonical:
In contrast, versions 0.8.3 and earlier will not catch this situation:
In order to preserve the earlier behavior, just submit require_public_global=False:
Handling Bad Data
Many CMS systems (and developers) create malformed content or incorrect document identifiers. When this happens, the BeautifulSoup parser will lose data or move it into an unexpected place.
There are two arguments that can help you analyze this data:
- force_doctype:
force_doctype=True will try to replace the identified doctype with “html” via regex. This will often make the input data usable by BS4.
- search_head_only:
search_head_only=False will not limit the search path to the “<head>” element. This will have a slight performance hit and will incorporate data from CMS/User content, not just templates/Site-Operators.
WARNING
1.0 will be a complete API overhaul. pin your releases to avoid sadness.
Version 0.9.19 Breaking Changes
Issue #12 exposed some flaws in the existing package
1. MetadataParser.get_metadatas replaces MetadataParser.get_metadata
Until version 0.9.19, the recommended way to get metadata was to use get_metadata which will either return a string (or None).
Starting with version 0.9.19, the recommended way to get metadata is to use get_metadatas which will always return a list (or None).
This change was made because the library incorrectly stored a single metadata key value when there were duplicates.
2. The ParsedResult payload stores mixed content and tracks it’s version———————————————————————--
Many users (including the maintainer) archive the parsed metadata. After testing a variety of payloads with an all-list format and a mixed format (string or list), a mixed format had a much smaller payload size with a negligible performance hit. A new _v attribute tracks the payload version. In the future, payloads without a _v attribute will be interpreted as the pre-versioning format.
3. DublinCore payloads might be a dict
Tests were added to handle dublincore data. An extra attribute may be needed to properly represent the payload, so always returning a dict with at least a name+content (and possibly lang or scheme is the best approach.
Usage
Until version 0.9.19, the recommended way to get metadata was to use get_metadata which will return a string (or None):
From an URL:
From HTML:
Malformed Data
It is very common to find malformed data. As of version 0.9.20 the following methods should be used to allow malformed presentation:
or:
The above options will support parsing common malformed options. Currently this only looks at alternate (improper) ways of producing twitter tags, but may be expanded.
Notes
when building on Python3, a static toplevel directory may be needed
Release historyRelease notifications | RSS feed
0.10.4
0.10.0
0.9.23
0.9.22
0.9.21
0.9.20
0.9.19
0.9.18
0.9.17
0.9.16
0.9.15
0.9.14
0.9.12
0.9.11
0.9.10
0.9.7
0.9.6
0.9.5
0.9.4
0.9.3
0.9.1
0.9.0
0.8.3
0.8.1
0.8.0
0.7.4
0.7.3
0.7.2
0.7.1
0.7.0
0.6.18
0.6.17
0.6.16
0.6.15
0.6.14
Meta Image 1 6 13 Esv
0.6.11
0.6.10
0.6.9
0.6.8
0.6.7
0.6.6
0.6.5
0.6.3
0.6.0
0.5.8
0.5.6
0.5.4
0.4.13
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size metadata_parser-0.10.4.tar.gz (45.3 kB) | File type Source | Python version None | Upload date | Hashes |
Hashes for metadata_parser-0.10.4.tar.gz
Algorithm | Hash digest |
---|---|
SHA256 | d427dfb3d0005d1dd78bdff2eb8bcfee23801e95e2b8fb58faf105ee78fc7531 |
MD5 | dcadd864af6d3d4d2316805b91da120c |
BLAKE2-256 | a19fcce147a839cecc5bf9aea8c2567ad91afaba0c808435a2596b9e0e285201 |
MetaImage supports more than 90 formats: PNG, JPEG and TIFF files as well as a large number of RAW files. You can contact us for more information or if the formats you need are not supported yet.
Notation: R: Read / W: Write / C : Create.
Notation: R: Read / W: Write / C : Create.
File Type | EXIF | IPTC | XMP | ICC | Other |
---|---|---|---|---|---|
3FR | R | R | R | R | - |
AI, AIT | R/W/C4 | R/W/C4 | R/W/C5 | R/W/C4 | R/W/C PDF PostScript, R Photoshop |
ARW | R/W/C | R/W/C | R/W/C | R/W/C | R/W Sony SonyIDC |
BPG | R | - | R | R | R BPG |
CR2 | R/W/C | R/W/C | R/W/C | R/W/C | R/W Canon, R/W/C CanonVRD |
CR3 | R/W/C | - | R/W/C | - | R/W Canon QuickTime, R/W/C CanonVRD |
CRM | R/W/C | - | R/W/C | - | R/W Canon QuickTime |
CRW, CIFF | - | - | R/W/C | - | R/W CanonRaw, R/W/C CanonVRD |
CS1 | R/W/C | R/W/C | R/W/C | R/W/C | R Photoshop |
DCM, DC3, DIC, DICM | - | - | - | - | R DICOM |
DCP | R/W/C | R/W/C | R/W/C | R/W/C | - |
DCR | R | R | R | R | - |
DNG | R/W/C | R/W/C | R/W/C | R/W/C | - |
DPX | - | - | - | - | R DPX |
DR4 | - | - | - | - | R/W/C CanonVRD |
EIP | R | - | - | - | R XML ZIP |
EPS, EPSF, PS | R/W/C | R/W/C | R/W/C | R/W/C | R/W/C PostScript, R Photoshop |
ERF | R/W/C | R/W/C | R/W/C | R/W/C | R/W Olympus |
EXIF | R/W/C | - | - | - | - |
EXR | - | - | - | - | R OpenEXR |
EXV | R/W/C | R/W/C | R/W/C | R/W/C | Supported JPEG Meta Information |
FFF | R/W/C | R/W/C | R/W/C | R/W/C | - |
FLIF | R/W/C | - | R/W/C | R/W/C | R FLIF |
FPF | - | - | - | - | R FLIR |
FPX | - | - | R | R | R FlashPix |
GIF | - | - | R/W/C | R/W/C | R/W/C GIF |
GPR | R/W/C | R/W/C | R/W/C | R/W/C | - |
HDP, WDP, JXR | R/W/C | R/W/C | R/W/C | R/W/C | - |
HEIC, HEIF | R/W/C | - | R/W/C | R/W | R/W QuickTime |
ICC, ICM | - | - | - | R/W/C | - |
IIQ | R/W/C | R/W/C | R/W/C | R/W/C | R/W PhaseOne |
IND, INDD, INDT | - | - | R/W/C | - | - |
INX | - | - | R | - | - |
J2C, J2K, JPC | R3 | R3 | R | R | R Jpeg2000 Photoshop |
JP2, JPF, JPM, JPX | R/W/C3 | R/W/C3 | R/W/C | R | R/W/C Jpeg2000, R Photoshop |
JPEG, JPG, JPE | R/W/C | R/W/C | R/W/C | R/W/C | Supported JPEG Meta Information |
K25 | R | R | R | R | - |
KDC | R | R | R | R | R Kodak |
MEF | R/W/C | R/W/C | R/W/C | R/W/C | - |
MIE | R/W/C | R/W/C | R/W/C | R/W/C | R/W/C MIE |
MIFF, MIF | R | R | R | R | R MIFF Photoshop |
MOS | R/W/C | R/W/C | R/W/C | R/W/C | R Leaf |
MPO | R/W/C | R/W/C | R/W/C | R/W/C | Supported JPEG Meta Information |
MRW | R/W/C | R/W/C | R/W/C | R/W/C | R/W MinoltaRaw Minolta |
NEF | R/W/C | R/W/C | R/W/C | R/W/C | R/W Nikon NikonCapture |
NRW | R/W/C | R/W/C | R/W/C | R/W/C | R/W Nikon NikonCapture |
ORF | R/W/C | R/W/C | R/W/C | R/W/C | R/W Olympus |
R3 | R3 | R/W/C | R3 | R/W/C PDF, R Photoshop | |
PEF | R/W/C | R/W/C | R/W/C | R/W/C | R/W Pentax |
PMP | - | - | - | - | R Sony |
PNG, JNG, MNG | R/W/C3 | R/W/C3 | R/W/C | R/W/C | R/W/C PNG |
PSD, PSB, PSDT | R/W/C | R/W/C | R/W/C | R/W/C | R Photoshop |
QTIF, QTI, QIF | R/W3 | R/W3 | R/W/C | - | R/W/C QuickTime |
RAF | R/W/C | R/W/C | R/W/C | R/W/C | R/W FujiFilm |
RAW | R/W/C | R/W/C | R/W/C | R/W/C | R/W PanasonicRaw Panasonic |
RW2 | R/W/C | R/W/C | R/W/C | R/W/C | R/W PanasonicRaw Panasonic |
RWL | R/W/C | R/W/C | R/W/C | R/W/C | R/W PanasonicRaw Panasonic |
RWZ | R | R | R | R | R Rawzor |
SR2 | R/W/C | R/W/C | R/W/C | R/W/C | R/W Sony |
SRF | R | R | R | R | R Sony |
SRW | R/W/C | R/W/C | R/W/C | R/W/C | R/W Samsung |
THM | R/W/C | R/W/C | R/W/C | R/W/C | Supported JPEG Meta Information |
TIFF, TIF | R/W/C | R/W/C | R/W/C | R/W/C | R/W/C GeoTIFF1, R/W Trailers |
VRD | - | - | R/W/C | - | R/W/C CanonVRD |
WEBP | R3 | - | R | - | R RIFF |
X3F | R/W/C | R/W/C | R/W/C | R/W/C | R/W Sigma, R SigmaRaw |
XCF | R | R | R | R | R GIMP |