> **Documentation Index:** Fetch the complete configuration index at https://yandex.com/dev/safebrowsing/doc/en/llms.txt # URL and hashing Safe Browsing lists consist of variable length SHA-256 hash prefixes (from 4 to 32 bytes) mapped to URLs of internet resources that pose threats to users. To see if a URL is on the list (in the local database or on the server), compute the hash prefix. To do this: 1. [Canonicalize the URL](#canonicalize). 1. [Create expressions for the host and path](#suffix-prefix). 1. [Compute the fully-qualified hash and the hash prefix](#hash). [Code sample in Golang](https://github.com/google/safebrowsing/blob/master/urls.go). Search the local Safe Browsing database. If you found a hash prefix, send it in your [request](https://yandex.com/dev/safebrowsing/doc/en/concepts/update-fullhashes-find.md). ## Canonicalized URL {#canonicalize} The URL must conform to [RFC 2396](https://tools.ietf.org/html/rfc2396). Make sure that you convert the URLs that use internationalized domain names (IDN), such as `.рф`, to ASCII using the [Punycode](https://en.wikipedia.org/wiki/Punycode) method. The URL must contain a path component. This means it must end with a slash, for example: `https://yandex.ru/`. To canonicalize a URL: 1. Delete the tab characters (`0x09`), carriage return characters (`0x0d`), and line feed characters (`0x0a`). 1. Delete the anchor. For example, reduce `https://yandex.ru/#anchor` to `https://yandex.ru/`. 1. Separately process: **Host** - Delete the leading and trailing dots. - Replace several consecutive dots with one dot. - If the host name is an IP address, convert it to four decimals, separated by a dot. - Make it lowercase. **Path**   : {% note info %} Don't apply these actions to the request parameters. {% endnote %} - Delete the `/../` sequences and replace `/./` with `/`. - Replace multiple consecutive slashes with one slash. 1. Escape all the characters with ASCII codes less than 32 and more than 127, as well as the `#` and `%` characters. Use uppercase hexadecimal notation. {% cut "Example" %} #| || **Source URL** | **Canonicalized URL** || || ``` http://host/%25%32%35 ``` | ``` http://host/%25 ``` || || ``` http://host/%25%32%35%25%32%35 ``` | ``` http://host/%25%25 ``` || || ``` http://host/%2525252525252525 ``` | ``` http://host/%25 ``` || || ``` http://host/asdf%25%32%35asd ``` | ``` http://host/asdf%25asd ``` || || ``` http://host/%%%25%32%35asd%% ``` | ``` http://host/%25%25%25asd%25%25 ``` || || ``` http://%32%31%33%2e%31%38%30%2e%32%31%30%2e5/%2e%73%65%63%75%72%65%2f%77%77%77%2e%6d%6f%69%6b%72%75%67%2e%72%75/ ``` | ``` http://213.180.210.5/.secure/www.moikrug.ru/ ``` || || ``` http://host%23.com/%257Ea%2521b%2540c%2523d%2524e%25f%255E00%252611%252A22%252833%252944_55%252B ``` | ``` http://host%23.com/~a!b@c%23d$e%25f^00&11*22(33)44_55+ ``` || || ``` http://3279880203/smth ``` | ``` http://195.127.0.11/smth ``` || || ``` http://www.yandex.ru/smth/.. ``` | ``` http://www.yandex.ru/ ``` || || ``` www.yandex.ru ``` | ``` http://www.yandex.ru/ ``` || || ``` http://www.zlo.com/smth#ancor ``` | ``` http://www.zlo.com/smth ``` || || ``` http://www.YANdex.ru/ ``` | ``` http://www.yandex.ru/ ``` || || ``` http://www.yandex.ru.../ ``` | ``` http://www.yandex.ru/ ``` || || ``` http://www.yandex.ru/m\ta\rp\ns ``` | ``` http://www.yandex.ru/maps ``` || || ``` http://zlo.com/smth#more#again ``` | ``` http://zlo.com/smth ``` || || ``` http://\x01\x80.com/ ``` | ``` http://%01%80.com/ ``` || || ``` http://www.hostport.com:1234/ ``` | ``` http://www.hostport.com/ ``` || || ``` http://www.yandex.ru/ ``` | ``` http://www.yandex.ru/ ``` || || ``` http:// probel.ru/ ``` | ``` http://%20probel.ru/ ``` || || ``` http://host.com//dvaslesha?more//slashes ``` | ``` http://host.com/dvaslesha?more//slashes ``` || |# {% endcut %} ## Expressions for the host and path {#suffix-prefix} After you canonicalize the URL, create: 1. Expressions for the host and path. 1. Combinations of those expressions. **Expressions for the host**   : Compose no more than 5 different strings: - A fully qualified host name. - Up to 4 host names, starting with the last five components and removing the first component sequentially. You can skip the top-level domain. Don't create those strings if the host name is an IP address. **Expressions for the path**   : Compose no more than 6 different strings: - Full path with the request parameters. - Full path without the request parameters. - 4 paths, starting from the root and sequentially adding path components with a trailing slash. **Combinations**   : Compose up to 30 different combinations of expressions for the host and path. Those combinations must include only the host name and path: remove the schema (protocol), username, password, and port. If the URL contains request parameters, at least one combination must contain the fully qualified path and request parameters. **Example**   : For the URL `http://a.b.c/1/2.html?param=1`, compose the following strings: ``` a.b.c/1/2.html?param=1 a.b.c/1/2.html a.b.c/ a.b.c/1/ b.c/1/2.html?param=1 b.c/1/2.html b.c/ b.c/1/ ``` For the URL `http://a.b.c.d.e.f.g/1.html`, compose the following strings: ``` a.b.c.d.e.f.g/1.html a.b.c.d.e.f.g/ (Skip b.c.d.e.f.g since only the last five host name components and the fully qualified host name are used) c.d.e.f.g/1.html c.d.e.f.g/ d.e.f.g/1.html d.e.f.g/ e.f.g/1.html e.f.g/ f.g/1.html f.g/ ``` For the URL `http://1.2.3.4/1/`, compose the following strings: ``` 1.2.3.4/1/ 1.2.3.4/ ``` ## Hashing {#hash} After you compose the host and path combinations, compute the full-length SHA256 hash for each of them. Then compute the prefix for each full-length hash. The hash prefix length is 4-32 bytes. Examples from the [FIPS-180-4](https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.180-4.pdf) standard: **Hashing of the `abc` string**   - The hash digest is `ba7816bf 8f01cfea 414140de 5dae2223 b00361a3 96177a9c b410ff61 f20015ad`. - The 32-bit prefix is `ba7816bf`. **Hashing of the `abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq` string**   - The hash digest is `ba7816bf 8f01cfea 414140de 5dae2223 b00361a3 96177a9c b410ff61 f20015ad`. - The 48-bit prefix is `248d6a61 d206`.