Remove logical_date check when validating inlets and outlets (#51464)

https://github.com/apache/airflow/commit/8865dbd4ca7d80aed7893688312e053c6d334022