Interacting particle systems are ubiquitous in nature and engineering. Access to the governing particle interaction law is of fundamental importance for a complete understanding of such systems. However, it is particularly challenging to extract this information from experimental observations due to the intricate configuration complexities involved. Machine learning methods have the potential to learn the behavior of interacting particle systems by combining experiments with data analysis methods. However, most existing algorithms focus on learning the kinetics at the particle level and do not learn the pairwise interactions specifically. Moreover, in reality, interacting particle systems are often heterogeneous, where multiple interaction types coexist simultaneously and relational inference is required. An approach that can simultaneously reveal the hidden pairwise interaction types and infer the unknown heterogeneous governing interactions constitutes a necessary advancement for our understanding of particle systems. However, this task is considerably more challenging than its homogeneous counterpart. Here, we propose the physics-induced graph network for particle interaction (PIG'N'PI) allowing to precisely infer the pairwise interactions that are consistent with underlying physical laws by only being trained to predict the particle acceleration for homogeneous systems. We further propose a novel method for relational inference which combines probabilistic inference and PIG'N'PI to learn different kinds of interactions for heterogeneous systems. We test the proposed methodologies on multiple benchmark datasets and demonstrate that the learnt interactions are consistent with the underlying physics and the proposed relational inference method achieves superior performance in correctly inferring interaction types. In addition, the proposed model is data-efficient and generalizable to large systems when trained on small systems, contrary to previously proposed solutions. The developed methodology constitutes a key element for the discovery of the fundamental laws that determine macroscopic mechanical properties of particle systems.