Just add the following lines to /etc/ntp.conf and restart ntp:

interface ignore wildcard
interface listen 127.0.0.1
interface listen ::1