I got it to work. But it's...UGLY.

To encode HDR correctly is very complex. HEVC will pass through 10 bit color options just fine, but you have to specify the HDR metadata in the encoder parameters or else HDR signaling will not work on most devices and it'll play back like an SDR file with washed out colors.

For files that use basic, static HDR (as in the HDR metadata is constant for the entire file), here's an example of what I use:

 -c:v libx265 -pix_fmt yuv420p10le -profile:v main10 -vsync 0 -preset slow 
 -level 5.1 -max_muxing_queue_size 4096 
 -x265-params crf=18:vbv-bufsize=11600:vbv-maxrate=5800:selective-sao=0:
  no-sao=1:strong-intra-smoothing=0:rect=0:aq-mode=1:hdr-opt=1:
  repeat-headers=1:colorprim=bt2020:range=limited:transfer=smpte2084:
  colormatrix=bt2020nc:range=limited:
  master-display="G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,50)":
  max-cll=1000,293

This string is what you should use for pretty much every HDR encode. The speed preset, vsync, CRF, buffersize, maxrate, and SAO settings are at your discretion -- these are just what I use for encodes to be streamed off my media server. One thing you DO need to pay attention to on EVERY HDR file is "max-cli=####,####" and the "L(10000000,50)" in the master-display definition. In the max-cll value, the first number is the Max-CLL, the second is Max-FALL. In the "L(10000000,50)" section, the first number is the max mastering display luminance value multiplied by 10,000 and the second is the minimum mastering display luminance value multiplied by 10,000. These values are all specific to EVERY HDR file and need to be set correctly or you may get noticeably wrong colors, brightness, and contrast in your encoded output. You can get these values by opening your source file in MediaInfo and looking at the bottom of the video metadata entry.

Now, for dynamic HDR (HDR10+ or Dolby Vision), it's even MORE involved - but there is indeed a way to create encoded files with dynamic HDR metadata now by using dovi_tool for Dolby Vision, or hdr10plus_tool for HDR10+ files (or you can use BOTH on it if you have a file that is compliant with both standards).

This requires de-muxing the dynamic metadata from the isolated HEVC videostream of your source file, encoding your file as an isolated HEVC stream, injecting the dynamic metadata back in to the encoded HEVC stream, then finally merging that with your MKV containing everything else.

Here's how to do it:

Step 1: extract your source file's raw video stream.

ffmpeg -i '\path\to\input\source_file.mkv' -v error -stats -c:v copy -an -vbsf hevc_mp4toannexb -f 
       hevc '\path\to\output\source_videostream.hevc'

Step 2: extract the file's dynamic HDR metadata to an RPU file (Dolby Vision) or a json file (HDR10+). Check the video stream metadata with mediainfo to see whether your input file uses Dolby Vision, HDR10+, or both, and run the corresponding commands:

dovi_tool -c -m 2 extract-rpu '\path\to\input\source_videostream.hevc' 
          -o '\path\to\output\DolbyVisionData.bin'

hdr10plus_tool extract '\path\to\input\source_videostream.hevc' 
               -o '\path\to\output\HDR10PlusData.json'

The dovi_tool command above is set up to crop out letterboxing and convert the Dolby Vision data to profile 8.1, which is the most widely supported DV profile. You can also set it up to rip the data without converting it between profiles, but support for those tends vary widely depending on the device you watch it on, and 8.1 is generally the most compatible.

Step 3: encode your file. Pay special attention to the bolded parts, which are additional settings you need to include in your encode script in order for the signaling to work correctly. If you do not need HDR10+ support, omit the "dhdr10-info='\path\to\HDR10PlusData.json'" option from the x265 encoder settings. If you do not need Dolby Vision support, omit the "dolby-vision-profile=8.1:dolby-vision-rpu='\path\to\DolbyVisionData.bin'" options from the x265 encoder settings.

ffmpeg -y -i '\path\to\input\source_file.mkv' -filter_complex '[0:v:0]zscale=1920:1080:filter=lanczos[1080]' -map '[1080]' -v error -stats -c:v libx265 -preset slow -pix_fmt yuv420p10le -profile:v main10 -vsync 0 -level 4.1 -max_muxing_queue_size 4096 -x265-params crf=18:vbv-bufsize=10800:vbv-maxrate=5400:selective-sao=0:no-sao=1:strong-intra-smoothing=0:rect=0:aq-mode=1:hdr-opt=1:dolby-vision-profile=8.1:dolby-vision-rpu='\path\to\DolbyVisionData.bin':dhdr10-opt=1:dhdr10-info='\path\to\HDR10PlusData.json':repeat-headers=1:colorprim=bt2020:range=limited:transfer=smpte2084:colormatrix=bt2020nc:range=limited:master-display='G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(Max mastering luminance times 10000,Min mastering luminance times 10000)':max-cll=<Max-CLL value>,<Max-FALL value> -vbsf hevc_mp4toannexb -f hevc '\path\to\encoded\videostream.hevc' -map_chapters 0 -metadata title='Example Title (year)' -map a:0 -c:a:0 ac3 -b:a:0 640k -ac:a:0 6 -metadata:s:a:0 language=eng -metadata:s:a:0 title='Surround 5.1' -map_metadata -0 -map a:0 -c:a:1 aac -b:a:1 256k -ac:a:1 2 -metadata:s:a:1 title='Stereo' -metadata:s:a:1 language=eng -map_metadata -1 -f matroska '\path\to\audio_chapters_headers.mka'

Note that the above script will output the videostream to an intermediate HEVC file and everything else to a separate mka file that you'll merge with your final videostream in a few steps.

Step 4: inject your Dolby Vision and/or HDR10+ metadata back into your encoded videostream.

For Dolby Vision:
dovi_tool inject-rpu -i '\path\to\encoded\videostream.hevc' --rpu-in '\path\to\DolbyVisionData.bin' 
          -o '\path\to\encoded\videostream-DV.hevc'

For HDR10+:
hdr10plus_tool inject -i '\path\to\encoded\videostream.hevc' -j '\path\to\HDR10PlusData.json' 
               -o '\path\to\encoded\videostream-HDR10+.hevc'

For BOTH Dolby Vision and HDR10+ (order of commands does not matter here):
hdr10plus_tool inject -i '\path\to\encoded\videostream.hevc' -j '\path\to\HDR10PlusData.json' 
               -o '\path\to\encoded\videostream-DV_HDR10+.hevc'

dovi_tool inject-rpu -i '\path\to\encoded\videostream.hevc' --rpu-in '\path\to\DolbyVisionData.bin' 
          -o '\path\to\encoded\videostream-DV_HDR10+.hevc'

Step 5: merge your final videostream into the file containing your audio, chapter, headers, etc:

mkvmerge -o '\path\to\final\outputfile.mkv' '\path\to\encoded\videostream-DV_HDR10+.hevc' 
         '\path\to\audio_chapters_headers.mka'

Or, if you prefer to merge the audio, chapters, etc from your source file:

mkvmerge -o '\path\to\final\outputfile.mkv' '\path\to\encoded\videostream-DV_HDR10+.hevc' 
         -D('\path\to\audio_chapters_headers.mka')

This will copy the HEVC stream into your merged mkv file, and copy everything EXCEPT the videostream from your source file into your merged file. Then just run a simple ffmpeg command to convert the finished MKV file to an MP4 file.

Good luck!