I recently installed MacOS High Sierra (10.13) on ESXi 6.0 for a developer I work with. The steps I used were from a number of sources found on the InterWebs.
First off you need a Mac to download the installer. On the Mac, go to the App Store and and download MacOS. While that is downloading, you’ll need to run the Unlocker script found at insanelymac.com. Please read the notes thoroughly, and yes, a reboot of the host is needed.
After the installer is downloaded, you will need to create the ISO. The following steps are all done on the Mac
Mount the installer:
hdiutil attach /Applications/Install\ macOS\ High\ Sierra.app/Contents/SharedSupport/InstallESD.dmg -noverify -nobrowse -mountpoint /Volumes/install_app
Then create a blank ISO
hdiutil create -o /tmp/HighSierra.cdr -size 7316m SPUD -fs HFS+J
Then mount the blank ISO
hdiutil attach /tmp/HighSierra.cdr.dmg -noverify -nobrowse -mountpoint /Volumes/install_build
Then restore the base image to the blank ISO
asr restore -source /Applications/Install\ macOS\ High\ Sierra.app/Contents/SharedSupport/BaseSysyem.dmg -target /Volumes/install_build -noprompt -noverify -erase
copy the install dependencies
cp /tmp/HighSierra.dmg /Volumes/OS\ X\ Base\ System/
unmount installer image
hdiutil detach /Volumes/OS\ X\ Base\ System
convert to iso
hdiutil convert /tmp/sierra.cdr.dmg -format UDTO -o /tmp/HighSierra.iso
rename to iso and place on Desktop
mv /tmp/HighSierra/iso.cdr ~/Desktop/HighSierra.iso
Enjoy on non Apple Hardware!